An Empirical Evaluation of MapReduce under Interruptions

The presence of interruptions is an unwanted but inevitable fact that all large-scale distributed computing systems have to face. The interruptions are more prevailed for MapReduce applications, as often MapReduce runs on the top of the commodity hardware based clusters, which are more vulnerable than traditional HEC systems. The problem is further exaggerated when running MapReduce applications in distributed nondedicated computing environment, where the host applications have the privilege to take back the computing power at random and interrupt MapReduce applications. This study intends to evaluate the resilience of MapReduce applications through an empirical evaluation. In particular, we set up a MapReduce system, inject interruptions with different patterns, and study their impact on the performance of the TeraSort benchmark. We simulate both cluster and distributed non-dedicated computing environment to observe the impact of these interruptions. Both the data locality and benchmark execution time have been measured. We also vary the number of replicas to observe its impact on the application performance. The experimental results show that interruptions have a significant impact on the performance of MapReduce applications. MapReduce in distributed computing environment is more vulnerable to interruptions due to the high data migration cost. Finally, we show that extra data replicas help to mitigate the impact of interruptions.